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ABSTRACT 



Healthy Start is a 5-year program that encourages 
community -based strategies for reducing infant mortality. This report 
addresses Congressional concerns of preliminary evaluation results reported 
by the press concerning the program’s efficacy. The report strives to answer 
three Congressional concerns: (1) the plan for the national program 

evaluation; (2) what the contractor hired to evaluate the plan's preliminary 
results has found thus far; and (3) what is expected from the final 
evaluation. The report concludes by noting that since the national evaluation 
of the Healthy Start program has yet to be completed, preliminary results 
should not be interpreted as conclusive. The final report on the national 
evaluation will include an extensive description of the program, indicate 
whether it has reduced infant mortality rates at Healthy Start sites, and 
provide an analysis of how program characteristics have influenced outcomes. 
The final evaluation report will analyze 4 years of data, though it is noted 
that this may be insufficient for judging the success of the program. 

Analysis of the fifth year will be done, but will not reflect as many years 
of mature program operation as possible; thus, a sixth-year evaluation is 
strongly suggested. (SD) 
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Over the past decade, the rate of infant mortality in the United States has 
steadily declined. In 1987, there were 10.1 deaths per 1,000 live births; by 
1996, the rate had dropped to 7.2 deaths. Yet, U.S. infant mortality rates 
have consistently been higher than those of many other developed 
countries. In addition, there are large racial differences in infant mortality 
rates in this country — in 1996, for example, the mortality rate for black 
infants was more than twice that for whites. Medical interventions can 
potentially address some of the leading causes of infant death. However, it 
is thought that poverty, inadequate community services, and educational 
factors prevent some women from gaining access to appropriate medical 
care. 

In an effort to reduce the nation’s infant mortality rate, in 1991 the Health 
Resources and Services Administration (hrsa) of the Department of Health 
and Human Services (hhs) initiated the Healthy Start program, which 
encourages community-based strategies for reducing infant mortality. The 
program was planned to be a 5-year demonstration project, and initially 15 
communities were awarded Healthy Start grants. 1 In 1996, the 
demonstration phase was continued for a sixth year. Although, the 
demonstration phase is considered to have been concluded, Healthy Start 
has continued into its seventh year. Over the life of the program, 48 
communities have been added to the original 15. Under the Healthy Start 
initiative, hrsa planned for an evaluation of the program to determine 
whether it had reduced infant mortality. In 1993, hrsa contracted with 
Mathematica Policy Research (mpr), Inc., to conduct a national evaluation 
of the 5-year demonstration in the original 15 sites. Although the 
evaluation is not complete, recent press reports have presented conflicting 



1 These sites were all or parts of Baltimore, Birmingham, Boston, Chicago, Cleveland, Detroit, the 
District of Columbia, New Orleans, New York, Northern Plains (19 Native American tribal 
organizations in four states), Northwest Indiana, Oakland, the Pee Dee Region of South Carolina, 
Philadelphia, and Pittsburgh. 
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stories on what mpr’s preliminary evaluation results indicate about Healthy 
Start. 

In light of these events, you asked us to (1) describe the plan for the 
national evaluation, (2) determine what mpr’s preliminary evaluation 
results indicate, and (3) describe what is expected from the final 
evaluation. To conduct our work, we reviewed the national evaluation’s 
preliminary results, available evaluation reports, and plans for future 
evaluations. We also interviewed program officials and mpr’s principal 
investigators and visited Philadelphia and Baltimore — 2 of the original 15 
Healthy Start communities recommended by hrsa as exemplifying 
alternative approaches to project organization. Other elements of the 
evaluation of Healthy Start, such as the local evaluations, are beyond the 
scope of our work. We conducted our study from January to May 1998 in 
accordance with generally accepted government auditing standards. 



Results in Brief 



mpr’s preliminary reports from the national evaluation of Healthy Start do 
not provide a complete assessment of the program and, therefore, should 
not be used to judge program success. Even the final report, likely to be 
delayed until early 1999 because of difficulties in acquiring vital record 
data from the states, will not contain all the data expected to be analyzed 
for the national evaluation. If the evaluation plan were expanded to 
include data from the sixth and final year of the demonstration, 
conclusions about whether the program has met its goal of reducing infant 
mortality could be strengthened. 

The national evaluation of the Healthy Start program has two major 
components: an impact evaluation, to determine whether infant mortality 
rates in Healthy Start communities have declined, and a process 
evaluation to describe how the program actually operates. Once these 
evaluations are completed, mpr plans to link outcomes with processes in 
its final report to determine why Healthy Start has or has not succeeded 
and what would be required for a similar intervention elsewhere. 

While mpr’s draft report on its impact evaluation suggests that Healthy 
Start has had little, if any, effect in reducing infant mortality in targeted 
communities, drawing such a conclusion at this time would be premature 
for several reasons. First, the results of the impact evaluation reflect the 
experience at only 9 of the 15 program sites to be evaluated. Second, the 
data are for only the first 3 of the 6 years of the demonstration. Third, 
since full implementation of the program took longer than anticipated at 
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most sites, the analyses reflect the earliest years of the program, before it 
was fully operational. 

The process evaluation is also incomplete. Only some of the reports that it 
comprises are available. Eventually, mpr plans to cover program 
implementation at all sites, the characteristics of program participants, 
and details about some of the most important strategies used by the 
program. With these two major components of the evaluation in 
preliminary stages or incomplete, mpr cannot yet relate process to impact. 

The final evaluation is expected to include an analysis of infant mortality 
data from the original 5 years of the demonstration for all 15 sites. 
However, the final report on the evaluation, now planned for early 1999, 
will include data from only the first 4 years. The fifth year will be included 
in an addendum to this report, to be submitted by mpr a year later. Further, 
because implementation of the program was slower than anticipated, and 
the program was mature for fewer years of the original demonstration 
period than planned for in the evaluation, even results from the “final 
report” are likely to be inconclusive and should be considered preliminary. 
While the fifth year of data to be reported in the addendum will be helpful, 
we are recommending that a sixth year of data be used to make the results 
more conclusive. 



Background 



In 1991, hrsa announced that it would fund 10 Healthy Start sites and 
issued guidance on how communities could obtain a grant. By July 1991, 
hrsa had received 40 applications, and in September of that year, it began 
funding 15 communities for a 5-year demonstration project. In 1996, 
funding for these communities was extended for a sixth year. In 1994, hrsa 
began funding seven new communities — called special projects — and 
funding for these was also extended in 1996 for an additional year. 
Forty-one additional communities have been awarded grants since 1997, 
and these now share funding with the 15 original sites and 5 of the special 
projects judged by hrsa to have been successful. 

To be eligible for the original grants, a community had to have an average 
annual infant mortality rate of at least 1.5 times the national average 
between 1984 and 1988 — that is, 15.7 deaths per 1,000 live births — and at 
least 50 but no more than 200 infant deaths per year. Applicants had to be 
local or state health departments, other publicly supported provider 
organizations, tribal organizations, private nonprofit organizations, or 
consortia of these organizations, hrsa required only a few specific 
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activities of all sites to provide grantees flexibility to make their projects 
relevant to local circumstances. 2 

Healthy Start’s principal goal to reduce infant mortality has usually been 
stated as a 50-percent reduction in infant mortality, attributable to the 
program, over 5 years. 3 Healthy Start also aims to achieve improvements 
in other outcomes — such as reductions in low birthweight, improved 
maternal health, and increased community awareness of threats to infant 
health — that are expected to help reduce infant mortality. In addition, 
Healthy Start was designed to demonstrate how a program based on 
innovation, community commitment and involvement, increased access to 
care, service integration, and personal responsibility could work in a 
variety of locations with high infant mortality. 

From fiscal year 1991 (a planning year that preceded the 5-year 
demonstration), through fiscal year 1998, program funding for Healthy 
Start has totaled more than $600 million. Healthy Start’s fiscal year 1992 
funding was less than half of what was initially proposed, and the number 
of grantees was greater. Instead of $171 million being spread over 10 sites, 
funding for the first year of the demonstration was $64 million spread over 
15 sites. 

In 1997, hrsa concluded the demonstration phase of Healthy Start and 
began the “replication phase” in 40 (now 41) new sites. In addition to 
providing Healthy Start services in their own communities, the established 
Healthy Start communities — the original 15 sites and 5 of the special 
projects — are mentoring several of the new sites. While the new sites 
receive, on average, somewhat less funding than the established sites, 
funding is shared among all sites. 



2 Grantees were to focus on reducing infant mortality, include the community in planning, assess local 
needs, increase public awareness of issues related to infant mortality, conduct a review of infant 
mortality in their area, develop innovative services for pregnant women and infants, and monitor their 
progress. 

3 HRSA officials acknowledge that the goal of 50-percent reduction was intended to be “motivational.” 
Other experts have asserted that this goal is not realistic. See D. Strobino and others, “A Strategic 
Framework for Infant Mortality Reduction: Implications of ’Healthy Start,’” The Milbank Quarterly, 
Vol. 73 (1995), pp. 507-33, in which it is demonstrated that the maximum reduction in infant mortality 
that possibly could be achieved as a result of a number of specific interventions, not all of which were 
implemented in all Healthy Start sites, is 32 percent 
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In September 1993, hrsa contracted with mpr to conduct the national 
evaluation of the Healthy Start program. This is currently funded with 
about $4.8 million, paid from the 1-percent set-aside for evaluation of 
health programs. 4 The original contract called for mpr to evaluate the first 
4 years of the 5-year demonstration program and contained an option for 
hrsa to request evaluation of the fifth year. In 1995, hrsa exercised that 
option, and the contract now requires that the evaluation cover all 5 years 
of the originally planned demonstration. Although the demonstration was 
extended for a year, hrsa currently has no plans to request that mpr 
evaluate the sixth and final year of the demonstration phase. 5 

The national evaluation, focused only on the original 15 sites, is designed 
to determine whether Healthy Start changed the rate of infant mortality 
and related outcomes, what factors contributed to any effects the program 
may have had, and how successful approaches to lessening infant 
mortality can be replicated in other communities. 6 Although each Healthy 
Start community is unique and the details of the delivery of any one 
service may differ across communities, many of the services are common 
to all sites: outreach and case management; support services, such as 
transportation and nutrition education; enhancements to clinical services; 
and public information campaigns. 

The national evaluation has two major components: an impact evaluation 
and a process evaluation. The impact evaluation is used to determine 
whether the infant mortality rates in Healthy Start communities have 
declined and whether related outcomes have improved. The process 
evaluation describes how the program actually operates. In its final 
evaluation report, mpr intends to synthesize these two components, linking 
outcomes with processes to determine why Healthy Start has or has not 
succeeded in communities and which strategies are likely to be successful 
elsewhere. 



The National 
Evaluation Consists of 
Both Impact and 
Process Evaluations 



4 P.L. 91-296 allows the Secretary of HHS to use up to 1 percent of the appropriations for programs 

authorized under the Public Health Service Act 

6 In addition to the efforts of MPR and its subcontractors, a 15-member Technical Advisory Group, 
consisting of experts in the fields of maternal and child health and program evaluation, provides 
consultation to MPR on all aspects of the design and conduct of the national evaluation. 

6 The original design for the national evaluation was prepared in 1992 by Lewin-ICF, Inc., and MDS 
Associates, Inc. It was reviewed by the 15 Healthy Start projects included in the demonstration and 
subsequently revised prior to issuance of the Request for Proposals. Early in the implementation of the 
national evaluation, the methodology was modified to accommodate changes in the availability of vital 
record data and the processes for which these and client data were obtained. 
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MPR’s Preliminary 
Evaluation Results 
Were Not Conclusive 



In the fall of 1997, mpr reported some preliminary evaluation results, 
including a draft interim report on its impact evaluation, which led to 
press accounts suggesting a variety of interpretations about the success of 
the Healthy Start program. We believe these preliminary evaluation results 
were not conclusive. Although the impact evaluation suggested that 
Healthy Start has not reduced infant mortality, such conclusions about the 
program would be premature because the impact evaluation does not 
include data from all the program sites or data from all the years of the 
program. Moreover, the process evaluation indicates that program 
implementation in many communities was slow and, therefore, that the 
impact data analysis may not be representative of a mature Healthy Start 
program. 



Preliminary Analysis of 
Program Impact May Not 
Be Meaningful Because of 
the Limited Data Available 



The national evaluation’s analysis of Healthy Start’s effect on infant 
mortality and related outcomes is preliminary — mpr characterized its 
October 1997 report as a draft. Because of problems obtaining data from 
some of the states’ departments of health, only 9 of the 15 program sites to 
be evaluated were represented in the analysis. In addition, the analysis is 
related to only the first 3 of the 6 years of program operation. Moreover, 
for illustrative purposes, mpr has limited its principal impact analysis to 
data from only the last of those 3 years, 1994. However, if, as hrsa believes, 
fiscal year 1995 was the first fully operational year, even 1994 data may not 
reflect the communities’ mature programs. 



To determine program impact, mpr is conducting two types of analysis: 
availability and participation. The availability analysis compares a Healthy 
Start community and two similar communities without Healthy Start to 
determine if the presence of the program in a community has an effect on 
infant mortality and related outcomes. The participation analysis 
compares, within a Healthy Start community, mothers who were clients of 
the program and mothers who were not. Both analyses can be used to 
study infant mortality; however, the availability analysis directly addresses 
the issue of reducing infant mortality in entire communities, while the 
participation analysis is restricted to outcomes for program participants. 

The national evaluation’s availability analysis found that for 1994, the 
overall infant mortality rate in Healthy Start communities was about the 
same as that in comparison communities. Applied to the individual sites, 
the analysis found that of the nine Healthy Start communities analyzed, 
only one experienced a significant reduction in infant mortality relative to 
its comparison sites, mpr similarly found that the neonatal and 
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postneonatal mortality rates — two components of infant mortality — were 
not significantly reduced in the Healthy Start communities relative to the 
comparison sites. 7 In its analysis of birth outcomes considered to be risk 
factors for infant mortality at eight of the Healthy Start communities, mpr 
found that in 1994, the low birthweight rate was reduced in only one 
community, the preterm birth rate was reduced in two other communities, 
and the rate at which women received adequate or better prenatal care 
was improved in five communities. 8 None of the analyses of data pooled 
from all sites yielded significant differences between sites with and 
without Healthy Start. 

The national evaluation’s participation analysis of Healthy Start’s effect on 
infant mortality has not been completed because of problems of data 
availability. The participation analysis of related outcomes, like the 
availability analysis, yielded little evidence of program effect in the eight 
communities analyzed. Participation in Healthy Start was not associated 
with reductions in low or very low birthweight rates or preterm birth 
rates. 9 In postpartum interviews with participants and nonparticipants, 
conducted in 1996, after all sites became fully operational, mpr found that 
participants were more likely to rate their prenatal care experience more 
highly and to be using birth control. However, no significant differences 
between participants and nonparticipants were reported for the receipt of 
services or health behaviors during pregnancy. 



Process Evaluation 
Indicates That Program 
Implementation Was More 
Gradual Than Anticipated 



The national evaluation’s process evaluation is intended to provide a 
detailed picture of what happened over time when Healthy Start was 
implemented at the various sites and assess its success in meeting its 
process objectives, such as hiring and retaining staff and putting the 
planned program in place. The evaluation, which, according to hrsa’s 
project officer for the evaluation, is to result in a series of reports, 
indicates thus far that the Healthy Start program was implemented largely 
as originally envisioned but more gradually than expected. 



7 The neonatal mortality rate is the number of deaths of infants under 28 days of age per 1,000 live 
births, and the postneonatal mortality rate is the number of deaths of infants between 28 days and 1 
year of age per 1,000 live births. 

8 The low birthweight rate is the number of live births weighing under 2,500 grams at birth per 100 live 
births. The preterm birth rate is the number of live births bom at less than 37 weeks of gestation per 
100 live births. The receipt of adequate or better prenatal care rate is the number of women who 
received “adequate” or “adequate plus” prenatal care, as defined by the Kotelchuck index of prenatal 
care, per 100 live births. See M. Kotelchuck, “An Evaluation of the Kessner Adequacy of Prenatal Care 
Index and a Proposed Adequacy of Prenatal Care Utilization Index,” American Journal of Public 
Health , Vol. 84 (1994), pp. 1414-20. 

9 The very low birthweight rate is the number of live births weighing under 1,500 grams at birth per 100 
live births. 
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mpr’s implementation report, a mgjor portion of the process evaluation, 
provides an overview of program implementation in 14 of the 15 original 
sites and draws conclusions about these projects. 10 The report includes 
detailed information on the development of the projects, the barriers to 
successful implementation, and the gaps between what was planned and 
what resulted. In addition, the report presents perceptions of variations 
across projects with respect to a variety of criteria, such as staff stability 
and consumer participation in the process. It also contains timelines 
indicating for each site when specific program components became 
operative. These timelines demonstrate, for example, that only 4 of the 14 
sites had all their planned services operational by October 1994. The 
implementation report concludes with lessons learned, which are 
organized into four categories: community context, organization and 
administration, community involvement, and service delivery. 

mpr has also completed a report on the infant mortality review process at 
the various Healthy Start sites. It indicates that, in general, the review 
programs are operational but with varying degrees of success in 
identifying the factors leading to infant mortality in their communities. 

Two detailed reports on specific interventions are available only in draft 
form. One describes, in greater detail than the implementation report, 
program participants, including comparisons of participants and 
nonparticipants with respect to the use of health and social services, 
satisfaction with services, and health-related behaviors, such as birth 
control and breast feeding. The other describes how outreach and case 
management were delivered, with consideration given to both similarities 
and differences across sites. 



The Degree to Which 
Specific Program 
Characteristics Contribute 
to Reductions in Infant 
Mortality Is Not Yet Known 



Because the impact and process evaluations are not finished, their 
synthesis has not yet begun, mpr expects to be able to draw conclusions 
about the program characteristics that are most effective in improving 
maternal and child health outcomes and the circumstances under which 
they are most likely to succeed when it integrates the impact and process 
evaluations. The synthesis of the impact and process components of the 
evaluation to be presented in the final report will be based on impact data 



10 E. M. Howell and others, The Implementation of Healthy Start Lessons for the Future (Washington: 
Mathematics Policy Research, Nov. 1997). A report on the project in the Northern Plains is to be 
published separately because of the grantee’s special circumstance as the only grantee serving a 
Native American population and its complex geographic and organizational structure, with 19 separate 
sites in four states. 
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from years one through four of the demonstration. This synthesis may 
have to be revised when more impact data are available. 



MPR’s Final Report 
Will Be Based on 
Fewer Years of Impact 
Data and Program 
Operation Than 
Originally Planned 



The final report as currently planned will not be the final evaluation of 
Healthy Start. The final report will contain an analysis of outcomes 
through 1995 and a synthesis of this with the findings of the process 
evaluation reports. Thus, it will assess the program’s impact on infant 
mortality through the fourth year of the demonstration, not the fifth year 
as planned. Further, these data will reflect the impact of only 1 or 2 years 
during which the program was fully operational. An addendum to the 
report, planned to follow a year later, will contain an updated analysis of 
outcomes through 1996. The addendum will assess impact on infant 
mortality through the fifth year of the demonstration and thus will reflect 
the impact of only 2 or 3 years during which the program was fully 
operational. 11 However, by evaluating the sixth year of the demonstration, 
it would be possible to obtain an analysis of 3 or 4 years of impact data 
from the mature program. 



Evaluating the sixth year of the demonstration would likely enhance the 
value of the investment in mpr’s evaluation for several reasons. First, 
including data from the sixth year of the demonstration would allow . 
evaluation of the years in which all 15 sites have been fully operational. 
Second, additional data would represent the effects of a more mature and 
potentially more effective program, which would likely provide more 
definitive answers about Healthy Start’s success. Third, having more years 
of data would increase the likelihood of detecting small but real effects of 
the program. Further, it is possible that data from the more mature years 
of the program will reflect program impact on the wider Healthy Start 
community, not just direct participants in program services and activities. 
In addition to hoped-for effects on the pregnant clients of Healthy Start, 
there may be effects of program services and education on those same 
women at other times, such as before or early in their next pregnancy; 
indirect effects on their social network, such as their male partners, 
friends, and sisters; and indirect effects on the community in general. 

hrsa’s project officer for the national evaluation notes that the cost 
associated with analyzing results for an additional year would be about 
$100,000; this would be inexpensive relative to the total national 



"According to HRSA officials, the agency anticipated that the data to assess the program’s impact on 
infant mortality might not be available within the period of the national evaluation. Consequently, in its 
Request for Proposals for the evaluation, HRSA asked for recommendations on how final year data 
might be acquired and analyzed. 
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evaluation cost of about $5 million. Since states collect vital records on 
births and deaths routinely, funds would be needed only to obtain, 
analyze, and report on the data and to revise the synthesis of these data 
with the process evaluation. 


Conclusions 


Since the national evaluation of the Healthy Start program has yet to be 
completed, preliminary results should not be used to conclude that the 
program has or has not achieved its goals, hrsa and mpr plan for the “final” 
report of the national evaluation to include an extensive description of the 
program, indicate whether it has reduced infant mortality rates at Healthy 
Start sites, and provide an analysis of how program characteristics have 
influenced outcomes. However, the final report will analyze infant 
mortality data from only 4 years of the demonstration. Primarily because 
implementation of the projects was slower than anticipated, data from the 
first 4 years of the demonstration may be insufficient forjudging the 
success of Healthy Start in lowering infant mortality. Thus, even the final 
report will be inconclusive. Analysis of the fifth year of the demonstration, 
as planned, will help strengthen the evaluation, but this analysis will not 
reflect as many years of mature program operation as possible. Thus, at a 
relatively modest cost, mpr’s evaluation would be further strengthened by 
including data from the sixth and final year of the demonstration. 


Recommendation 


To increase the value of the investment in the national evaluation of 
Healthy Start, we recommend that hrsa contract with mpr to expand the 
evaluation to include impact data from the sixth year. 


Agency Comments 


In commenting on a draft of this report, hrsa agreed with our findings and 
indicated that it intends to add funds to the mpr contract to include impact 
data from the sixth year of the demonstration, hrsa and mpr provided a 
number of technical comments that we incorporated as appropriate. 




As arranged with your office, unless you announce its contents earlier, we 
plan no further distribution of this report until 30 days after the date of 
this letter. We will then send copies to the Secretary of Health and Human 
Services, to the Administrator of hrsa, and to others who are interested. 
We will also make copies available to others on request. 
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Please contact me at (202) 512-7119 if you or your staff have any questions. 
You may also contact Michele Orza, Assistant Director, at 512-9228, or 
Donald Keller, Senior Evaluator, at 512-2932. 




Bernice Steinhardt 
Director, Health Service Quality 
and Public Health Issues 
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MPR’s Methodology and Its Limitations 



The Availability 
Analysis 



The availability analysis of infant mortality and related birth outcomes is 
part of Mathematica Policy Research’s (mpr) attempt to determine if 
Healthy Start has, as intended, reduced infant mortality at program sites, 
looking at the vital statistics for entire program and comparison areas 
where, respectively, the program is or is not available. It does this without 
concern about the participation in the program of specific persons. It 
attempts to separate any change that may occur in outcomes at program 
sites that is attributable to Healthy Start from change in outcomes at those 
sites that would have occurred without the program — for example, 
changes stemming from the national trends not related to health and social 
interventions, such as the persisting decline in infant mortality 
experienced almost everywhere in the United States. It does this for each 
outcome of interest, obtained from the state health department’s vital 
records of linked births and deaths for the sites of interest, by 
(1) comparing each program site with two comparison sites without 
Healthy Start, selected (matched) for similarity to the program site with 
respect to race and ethnicity, infant mortality rate, and trend in infant 
mortality over the pre-Healthy Start period and (2) statistically adjusting 
the data for differences between program and comparison site mothers on 
variables, also obtained from vital records, believed to affect the outcome. 
To the extent that, as a result of site selection and statistical adjustment, 
the program and comparison sites do not differ in expected infant 
mortality rate, then the comparison of the program and comparison site 
adjusted outcomes should be a valid indication of the effectiveness of the 
program. 

mpr’s approach involves accepted statistical methods with known 
limitations. One limitation stems from the possibility that program and 
comparison site mothers will systematically differ in ways, such as poverty 
level, that affect outcomes but are not taken into account in the selection 
of comparison sites and are not available for use in statistically adjusting 
the data. Such a difference could bias the estimation of the difference 
between program and comparison sites in outcomes. Nevertheless, mpr 
appears to have taken reasonable precautions to minimize the likelihood 
of bias, mpr did this, for example, by using two comparison sites, not just 
one, for each program site and by avoiding the selection of comparison 
areas known to have interventions similar to Healthy Start. Further, mpr 
shared information on its site selections with each of the 15 sites and 
sought their comments and agreement on the choices. 

Another potential limitation of the analysis concerns its statistical power, 
the ability to detect differences between program and comparison site 
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reductions when such differences, in fact, exist. Roughly speaking, power 
depends on the number of observations — in this case, live births — in the 
analysis, and it is therefore often a problem when that number is not 
controlled by the design of the study. In the case of Healthy Start, this 
implies that the ability to detect a real difference between program and 
comparison communities depends upon whether the comparison involves, 
for example, communities relatively small in population, large 
communities, or all communities pooled. With respect to infant mortality, 
mpr reports that, using all data from 1984 to 1994, the minimal detectable 
difference in infant mortality is computed to be 31 percent, 7 percent, and 
6 percent for small, large, and all communities, respectively. This means 
that if there are real differences between Healthy Start and comparison 
communities, but these differences are smaller than we have the power to 
detect (that is, smaller than the percentages listed above), then we will 
mistakenly conclude that the program has no effect on infant mortality. 
Since power depends on the number of observations, increasing the 
number of years of data included in the analysis will increase the ability to 
detect any difference that may exist. 

A third potential limitation concerns the number of statistical tests 
performed in the complete impact analysis. If 15 sites and seven different 
birth-related outcome variables are considered, then at least 105 statistical 
tests will be done. With as large a number of tests as this being done, it is 
likely that a portion of them will yield statistically significant results by 
chance alone. This means that even if Healthy Start has no effect on infant 
mortality, we will mistakenly conclude that it does have one in a certain 
percentage of the statistical tests conducted. There are statistical methods 
of dealing with this problem. If they are not employed in the final analysis, 
then differences that are statistically significant by chance alone will occur 
more often than is considered acceptable by statistical convention. 



The Participation 
Analysis 



mpr’s participation analysis of birth outcomes is part of mpr’s attempt to 
determine the effect of participation in Healthy Start within program sites. 
It compares 1995 birth outcomes between participants and 
nonparticipants in each project area. Participants in the program’s 
prenatal activities are identified from program files, the Minimum Data Set 
required of all Healthy Start sites, and their birth certificates are flagged. 
Their birth outcomes are then compared with those of nonparticipants or 
participants with limited prenatal program involvement. 
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This kind of analysis is limited by possible preexisting differences between 
participants and nonparticipants and by the very definition of participant. 
Since participation in Healthy Start is voluntary, it is possible that 
participants systematically differ from nonparticipants. Program providers 
may, for example, tend to attract persons who are especially 
knowledgeable about services or already well connected to the health care 
system. Under these circumstances, program participants would be 
expected to have better outcomes even without Healthy Start. 

Alternatively, participants might be especially needy and at high risk for 
poor outcomes, in which case they would be expected to have relatively 
poor outcomes, mpr deals with this by statistically adjusting the data on 
the basis of information that may reflect these preexisting differences, 
background information from birth certificates, and any other available 
sources. Although it is difficult to be certain that outcomes have been 
adjusted for all possible systematic differences between the groups being 
compared, mpr has stated that participants tend to be at high risk for poor 
birth outcomes, thereby making any potential finding of better outcomes 
for them than for nonparticipants more convincing of the program’s value. 

The question of who is a participant must be answered in order to conduct 
the participation analysis. It turns out not to be easily answered because 
(1) the Minimum Data Sets of many sites have been slow in developing 
into accurate record systems, (2) it is not always clear whether a 
participant’s involvement has been intense enough to classify that person 
as a participant, and (3) when supplementary information has been sought 
from new mothers about their involvement in the program it is not always 
clear what criteria they use forjudging whether or not to claim to be 
participants. Moreover, these problems vary somewhat from site to site, 
making it difficult to be sure that all participation analyses are 
comparable. 

In addition to the participation analysis of infant mortality and birth 
outcomes, mpr analyzes the results of its survey of postpartum mothers in 
Healthy Start areas by comparing program participants with 
nonparticipants. Most of the survey items concern service use, satisfaction 
with care, behaviors like practicing birth control and breast feeding, and 
ratings of infant health. Some of these “intermediate outcomes” may be 
associated with infant mortality and others may be more closely related to 
topics from the process evaluation. Either way, however, the limitations 
that apply to the participation analysis of infant mortality and birth 
outcomes also apply to these participation analyses. The results will be 
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meaningful to the extent that the preexisting differences between 
participants and nonparticipants can be taken into account. 



The Process 
Evaluation 



mpr’s process evaluation is an effort to use both qualitative and 
quantitative information to assess the degree to which Healthy Start has 
implemented its program as conceived of, how it serves its target 
population, and how these processes developed over time. This 
description of Healthy Start can be considered the documentation of the 
program’s second goal — to demonstrate what happens when this kind of 
effort is mounted. Its methods are varied, including making site visits with 
and telephone calls to project staff, examining the client records of the 
Minimum Data Set, the postpartum survey of participants and 
nonparticipants, running focus groups with service providers and with 
members of the communities, as well as using documents and vital 
records. Many aspects of the process evaluation are not complete and will 
not therefore be described further, but one major document — the 
implementation report — is complete. 

The implementation report is based mainly on site visits, expenditure 
reports from each project, and the client records of the Minimum Data Set. 
Further, two independent teams of site visitors rated certain dimensions of 
administrative success using a modified Delphi consensus reaching 
process. Although they may not be avoidable, the limitations of this report 
are those common to most process evaluations that are heavily qualitative. 
The methods employed provide a wealth of information suitable to inform 
those who would develop similar programs about what to expect if 
different options of organization, administration, and mode of service 
delivery are attempted. However, the essential subjectivity of interview 
methods makes it difficult to know how closely other evaluators would 
agree with the conclusions drawn. 
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